<html>
<head>
  <title>Linear Regression</title>
  <basefont face="微软雅黑" size="2" />
  <meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
  <meta name="exporter-version" content="Evernote Windows/307027 (zh-CN, DDL); Windows/6.1.0 (Win32);"/>
  <style>
    body, td {
      font-family: 微软雅黑;
      font-size: 10pt;
    }
  </style>
</head>
<body>
<a name="952"/>
<h1>Linear Regression</h1>

<div>
<span><div><div><h1><span style="font-size: 18pt; color: rgb(28, 51, 135); font-weight: normal;">一般线性回归遇到的问题</span></h1><ul><li><span style="font-size: 11pt;">预测精度：这里要处理好这样一对为题，即样本的数量<img src="Linear Regression_files/Image.gif" type="image/gif" data-filename="Image.gif"/>和特征的数量<img src="Linear Regression_files/Image [1].gif" type="image/gif" data-filename="Image.gif"/></span></li><ul><li><span style="font-size: 11pt;"><img src="Linear Regression_files/Image [2].gif" type="image/gif" data-filename="Image.gif"/>时，最小二乘回归会有较小的方差</span></li><li><span style="font-size: 11pt;"><img src="Linear Regression_files/Image [3].gif" type="image/gif" data-filename="Image.gif"/>时，容易产生过拟合</span></li><li><span style="font-size: 11pt;"><img src="Linear Regression_files/Image [4].gif" type="image/gif" data-filename="Image.gif"/>时，最小二乘回归得不到有意义的结果</span></li></ul><li><span style="font-size: 11pt;">模型的解释能力：如果模型中的特征之间有相互关系，这样会增加模型的复杂程度，并且对整个模型的解释能力并没有提高，这时，我们就要进行特征选择。</span></li></ul><div> 以上这些问题，<span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">主</span><span style="font-size: 12pt; color: rgb(173, 0, 0); font-weight: bold;">要就是表现在模型的方差和偏差问题上，方差指的是模型之间的差异，而偏差指的是模型预测值和数据之间的差异，我们需要找到方差和偏差之间的折中。</span></div></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><span style="color: rgb(28, 51, 135); font-size: 18pt;">岭回归的概念</span></div><div><br/></div><div><font style="font-size: 14pt;"><span style="font-size: 14pt; font-weight: bold;">特征选择方式</span></font></div><div><ul><li><span style="line-height: 1.45;">子集选择</span></li><li><span style="font-size: 11pt;">收缩方式(Shrinkage method)，又称为正则化(Regularization)。主要包括岭回归个lasso回归</span></li><li>维数减缩</li></ul></div><div><span style="font-size: 11pt;">岭回归(Ridge Regression)是在平方误差的基础上增加正则项</span></div><div style="text-align: center; margin-top: 1em; margin-bottom: 1em;"><div><span style="font-size: 11pt;"><img src="Linear Regression_files/Image [5].gif" type="image/gif" data-filename="Image.gif"/>,<img src="Linear Regression_files/Image [6].gif" type="image/gif" data-filename="Image.gif"/></span></div></div><div><span style="font-size: 11pt;">通过确定<img src="Linear Regression_files/Image [7].gif" type="image/gif" data-filename="Image.gif"/>的值可以使得在方差和偏差之间达到平衡：随着<img src="Linear Regression_files/Image [8].gif" type="image/gif" data-filename="Image.gif"/>的增大，模型方差减小而偏差增大。</span></div><div><br/></div><div>岭回归用于处理下面两类问题：</div><div style="margin-top: 1em; margin-bottom: 1em;">1.数据点少于变量个数</div><div style="margin-top: 1em; margin-bottom: 1em;">2.变量间存在共线性</div><div>变量间存在共线性是，最小二乘回归得到的系数不稳定，方差很大，这是因为系数矩阵x与它的转置矩阵相乘得到的矩阵不能求逆，而ridge regression通过引入lamda参数，使得该问题得到解决。</div><div><img src="Linear Regression_files/Image.png" type="image/png" data-filename="Image.png"/>              <img src="Linear Regression_files/Image [1].png" type="image/png" data-filename="Image.png"/></div><div>This is an example of <span style="font-weight: bold;">bias/variance tradeoff</span>: the larger the ridge alpha parameter, the higher the bias and the lower the variance.</div><div><br/></div><h3>Sparsity</h3><div>        A representation of the full diabetes dataset would involve 11 dimensions (10 feature dimensions and one of the target variable). It is hard to develop an intuition on such representation, but it may be useful to keep in mind that it would be a <span style="font-weight: bold; color: rgb(173, 0, 0);">fairly</span> <span style="font-weight: bold; color: rgb(173, 0, 0); font-style: italic;">empty</span> <span style="font-weight: bold; color: rgb(173, 0, 0);">space（相当空的n维空间）.</span></div><div><br/></div><div>        select only the informative features and set non-informative ones, like feature 2 to 0. Ridge regression will decrease their contribution, but not set them to zero. Another penalization approach, called <a href="http://scikit-learn.org/stable/modules/linear_model.html#lasso">Lasso</a> (least absolute shrinkage and selection operator), can set some coefficients to zero. Such methods are called <span style="font-weight: bold;">sparse method</span></div><div><br/></div><h3>Classification</h3><div>        For classification, as in the labeling <a href="https://en.wikipedia.org/wiki/Iris_flower_data_set">iris</a> task, <font style="font-size: 12pt;"><span style="color: rgb(173, 0, 0); font-size: 12pt; font-weight: bold;">linear regression is not</span></font> the right approach as it will give too much weight to data far from the decision frontier. A linear approach is to fit a sigmoid function or <span style="font-weight: bold;">logistic</span> function</div><div><font style="font-size: 12pt;"><span style="color: rgb(173, 0, 0); font-size: 12pt; font-weight: bold;">T</span></font><font color="#AD0000" style="font-size: 12pt;"><span style="color: rgb(173, 0, 0); font-size: 12pt; font-weight: bold;">his is known as</span> <a href="http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html#sklearn.linear_model.LogisticRegression" style="font-size: 12pt; font-weight: bold; color: rgb(28, 51, 135);" title="sklearn.linear_model.LogisticRegression">LogisticRegression</a><span style="color: rgb(173, 0, 0); font-size: 12pt; font-weight: bold;">.</span></font></div><div><br/></div><div><font style="font-size: 14pt;"><span style="font-size: 14pt; font-weight: bold;">Multiclass classification</span></font></div><div><br/></div><div>        If you have several classes to predict, an option often used is to fit one-versus-all classifiers and then use a voting heuristic for the final decision.</div><div><br/></div><div><br/></div><div><br/></div><h2><span style="color: rgb(28, 51, 135); font-size: 18pt; font-weight: normal;">Support vector machines (SVMs)</span><span style="line-height: 1.45;">  </span></h2><h3>Linear SVMs</h3><div>         <span style="line-height: 1.45;">SVMs can be used in regression –</span><a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html#sklearn.svm.SVR" style="line-height: 1.45;" title="sklearn.svm.SVR">SVR</a> <span style="line-height: 1.45;">(Support Vector Regression)–, or in classification –</span><a href="http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC" style="line-height: 1.45;" title="sklearn.svm.SVC">SVC</a> <span style="line-height: 1.45;">(Support Vector Classification).</span></div><div>        <font style="font-size: 12pt;"><span style="color: rgb(227, 0, 0); font-size: 12pt; font-weight: bold;">Regularization</span></font> is set by the C parameter: a small value for C means the margin is calculated using many or all of the observations around the separating line (more regularization); a large value for C means the margin is calculated on observations close to the separating line (less regularization).</div><div><br/></div><h3>Using kernels</h3><div><span>    <span>    </span></span>Classes are not always linearly separable in feature space. The solution is to build a decision function that is not linear but may be polynomial instead. This is done using the <span style="font-style: italic;">kernel trick</span> that can be seen as creating a decision energy by positioning <span style="font-style: italic;">kernels</span> on observations</div><div><img src="Linear Regression_files/Image [2].png" type="image/png" data-filename="Image.png"/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div><div><br/></div></div><div><br/></div></span>
</div></body></html> 